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ABSTRACT 



As the University of Arizona implemented its curriculum 
reform initiative, faculty became much more planful about their teaching, but 
also much more demanding of credible advice about instruction. Faculty make 
sometimes paradoxical demands in these circumstances: demands for tools that 
make it easy to do new things, and demands that these tools be accompanied by 
documentation of their effectiveness that can only be obtained after the 
tools are put to use. In order to respond to these demands, the main strategy 
was to build a learning community among faculty, using an online 
instructional support system, POLIS (Project for On-Line Instructional 
Support) , both as a platform for dissemination of new ideas about teaching 
and learning and as a device for continuous collection of data on strategies 
and outcomes. The current version of the system is available on the World 
Wide Web; it is both a production site for University of Arizona courses and 
a demonstration site for public use. It is a Web course construction kit that 
allows instructors to build a course on the Web for immediate delivery to 
students over the Web. This paper describes POLIS, how POLIS disseminates 
advice about teaching, and how POLIS gathers data on teaching and learning. 
Contains 11 references. (AEF) 
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S tarting about four years ago, the University of Arizona began a major initiative to try to reform teaching practice 
and to rebuild serious scholarly interest in teaching. The central component of this initiative was reform of the 
general education curriculum, which has now been completed. But an important offshoot of the initiative was a re- 
examination of teaching and its importance relative to other areas of faculty responsibility. Tenure and post-tenure review 
procedures were rewritten to weight teaching more heavily in evaluation and to require more documentation of teaching 
effectiveness and more explicit defense of teaching choices. Workshops and symposia were offered on such instructional 
designs as cooperative and collaborative learning. Study groups were formed to try to figure out how the university could 
become more ‘student-centered.’ Grants were awarded across campus for competitively selected instructional innovation 
projects. Gradually, the idea spread that there might be a need for change in our routine teaching practices, and much 
interest began to center on the possibilities presented by the Worldwide Web and other electronic media. 



In this changed institutional environment, faculty are 
becoming much more planful about their teaching but also 
much more demanding of credible advice about teaching. 
Urged by instructional designers and information technol- 
ogy specialists to experiment with educational use of the 
Worldwide Web, faculty raise challenging questions about 
the value of new technology and about the balance 
between effort and return. They want to know whether 
investing in development of materials for the Web will pay 
off in any sense: whether it will improve learning, and if 
so, whether the improvement will be substantial enough to 
compensate for the time invested in development and for 
resistance from students. Many have heard from colleagues 
of the frustrations of building elaborate websites for classes 
that students never visit or, worse, visit but regard as 
useless. 

Even faculty who are intent on improvement are often 
leery of change. This paper documents an effort to respond 
to the sometimes paradoxical demands faculty make in 
these circumstances: demands for tools that make it easy to 
do new things, and demands that these tools be accompa- 
nied by documentation of their effectiveness that can only 
be obtained after the tools are put to use. Our main strategy 
has been to try to try to build a learning community among 
faculty, using an online instructional support system both 
as a platform for dissemination of new ideas about 
teaching and learning and as a device for continuous 
collection of data on strategies and outcomes. 



The Support System: POLIS 

POLIS stands for “Project for On-Line Instructional 
Support,” and the current version of the system is available 
on the Worldwide Web both as a production site for 
University of Arizona courses and as a demonstration site 
for public use (Jackson, 1996). POLIS is a web course 
construction kit that lets instructors build a course on the 
Worldwide Web in any subject, for immediate delivery to 
students over the Worldwide Web. No special training is 
required to use POLIS. 75 distinct courses were mounted 
on POLIS for Fall 1997, despite a complete absence of 
training programs or advertising. Instructors found the 
tools through word of mouth, search of the University’s 
website, or direction to POLIS from the University 
Teaching Center. Most instructors never need assistance of 
any kind in using the system. 

To build a course, the instructor visits a certain website 
and creates interactive web pages using POLIS resources. 
The resources include things like student-built bibliogra- 
phy and webliography, support tools for student study 
groups and student project teams, and several varieties of 
discussions. Instructors can also upload their own 
hypertext or multimedia content and link or embed these in 
POLIS resources, or they can use POLIS tools from within 
an independently maintained website. Although there is 
considerable flexibility within POLIS, a beginner can 
create an entire course without dealing with any of the 
complexities normally associated with web publishing. 
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The ease with which POLIS can be used is an impor- 
tant design feature of the system, but it is not the most 
important feature of POLIS. The most important feature of 
POLIS is that it allows for creation of highly sophisticated 
interactive dialogues on any topic, again without requiring 
any web programming skills. All resources in POLIS, but 
especially the interactive dialogues, aim for “significant” 
interaction requiring thought and effort (Milheim, 1995- 
96), mostly in the form of free verbal resource to questions 
and arguments. 

At present, POLIS offers four well-defined dialogue 
types (in addition to a general-purpose threaded discussion 
format). Known as “POLIS lesson protocols,” these 
dialogue types are online simulations of interaction 
sequences known from prior research to be effective in 
traditional classroom settings. Each online protocol can be 
described as a sequence of display and response elements 
(Jackson & Madison, 1997). 

One-Minute Essay. In traditional classroom settings, 
the one-minute essay involves simply having students 
write for a very short time on a topic covered in the lecture 
or discussion, sometimes as a way to develop materials 
prior to discussion and sometimes to summarize, integrate, 
or reflect on materials already covered (refs). The online 
version has two required elements (a prompt and a student 
response) that can be elaborated with an optional display 
of student writings, shown before or after the student’s 
own submission. As compared with the inclass version, the 
online version has two potential advantages: its adaptabil- 
ity for asynchronous interaction and its automatic 
archiving and publication of student writings. 

Recitation. A standard recitation is a question/answer/ 
assessment sequence. Although this familiar protocol has 
fallen out of favor as constructivist ideas have taken hold, 
it remains a popular tutorial sequence. The POLIS online 
recitation has four elements: the question, the student’s 
answer, a “model answer” to which the student’s own 
answer can be compared, and a self-assessment. Like the 
One-Minute Essay, the Recitation protocol can include 
optional elements such as display of any amount of 
background information in text or multimedia, and it can 
include display of other students’ writings at any point in 
the response sequence. As compared with an inclass 
recitation, the online version has the advantages mentioned 
above, plus the advantage of allowing every student to 
give an independent answer to every question in a set of 
exercises. 

Adversary. Responding to many recent calls for an 
increased reliance on argumentation in learning, especially 
in the sciences (Kuhn, 1993; Kuhn, Shaw, & Felton, 1997; 
Pontecorvo, 1993; Meyer & Woodruff, 1997; Zeidler, 
1997), POLIS offers a simulated online debate protocol in 
which students are invited to stake out positions and 
defend them against counterarguments written by the 
instructor or selected from other students’ writings. The 



Adversary protocol’s elements are as follows: description 
of a controversy; student selection of a standpoint and 
preliminary defense of that standpoint; presentation of an 
opposing argument; student reaction to opposing argu- 
ment; and student reconsideration of standpoint and 
explanation for changing standpoints or standing pat. Any 
number of opposing arguments can be presented for the 
student’s reaction. The next revision of this protocol will 
also allow for review of other students’ arguments on 
either side of the issue. 

Virtural Peer. Based on Mazur’s (1997) ConcepTest 
protocol, the Virtual Peer is an alternative format for 
incorporation of argumentation into courses in any subject. 
Mazur’s protocol, as used in traditional classroom settings, 
has 6 distinct elements: presentation of a problem with a 
set of candidate answers; silent reflection by students; 
commitment by each individual student to one of the 
candidate answers; argumentation among pairs of students 
trying to convince one another of the correctness of then- 
answers; reconsideration of candidate answers; discussion 
of the correct solution and reasons why other candidate 
answers are incorrect. The online version presents the 
problem with a set of candidate answers; asks for the 
reasoning leading to the selected answer; presents contra- 
dictory reasoning favoring each unchosen alternative and 
asks how the student would respond to a classmate 
reasoning in this way; presents the candidate answers for 
reconsideration and asks for reasoning supporting the 
student’s new answer; and finally presents the correct 
solution. The online version sacrifices live interaction with 
peers, but improves on the live version by assuring that 
every student will have to respond to reasoning that differs 
from his or her own and by assuring that this reasoning 
will present significant intellectual challenges. 

How POLIS Disseminates Advice About 
Teaching 

When instructors open their lesson composer in POLIS, 
they are presented with choices among four supported 
protocols, and along with structural descriptions of each 
protocol they see a short passage containing “Recommen- 
dations.” The recommendations give a very short summary 
of the research basis for the protocol, explaining its 
origination and whether it has been shown to enhance 
learning. For example, the recommendations accompany- 
ing the Virtual Peer protocol read as follows: “No-tech 
inclass version has been shown to be very effective in 
teaching both science concepts and problem-solving. 

Online version allows for close control over the ‘peer’ 
contributions to learning.” The recommendations accom- 
panying the One-Minute Essay protocol poin: cut that the 
contributions of routine writing to learning do not depend 
on whether an instructor grades each piece of writing. 

Instructors come to POLIS as an easy way to create 
interactive course materials on the web. However, their 
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actual engagement with POLIS has the potential for more 
profound and pervasive change in teaching practice, and 
this potential follows as much from the design of the 
authoring tools as from the design of the protocols 
themselves. Two features of the design of the authoring 
system are significant for the present discussion. 

First, the display of four distinct protocols, each 
described in terms of its display and response elements, 
draws attention to a more abstract process of lesson 
planning. The four protocols do not exhaust all of the 
possible teaching/leaming dialogues that can be composed 
once an instructor begins thinking strategically about forms 
of interactional engagement. The recommendations draw 
attention explicitly to the features of each protocol, and 
more importantly, they draw attention to the possibility of 
using these protocols in traditional classroom settings as 
well as in online applications. So the lesson construction 
tool is not just an authoring tool, but also a learning 
environment for teachers, a place where they are intro- 
duced to a relatively unfamiliar way of thinking about 
lesson planning that can be assimilated into practice in 
many ways. 

Second, the recommendations emphasize the theoreti- 
cal and empirical justification for each protocol. The 
recommendations explain what grounds there are for 
believing that these protocols will produce effective 
lessons. The inclusion of a protocol in the POLIS lesson 
composer is a sort of institutional stamp of approval for the 
use of these kinds of interactions in teaching. Protocols 
that have produced negative results in research (Marttunen, 
1992) are not supported within POLIS. However, even for 
protocols that have been included, the stamp of approval is 
clearly provisional, subject to revision as experience with 
the online protocols either confirms or contradicts experi- 
ence using similar protocols in traditional classroom 
settings. The design of the authoring environment is meant 
to emphasize that planful teaching has grounding in 
research and reflection. 

POLIS is officially a support system for online 
instruction, meant to assist instructors in moving 
coursework onto the web. It should be obvious, however, 
that the support tools and the learning environment they 
create have the potential to alter the way instructors plan 
for face-to-face meetings with their students. 

How POLIS Gathers Data on Teaching 
and Learning 

We mentioned earlier that POLIS does not simply 
disseminate what is currently known about teaching and 
learning, but also allows us to continuously accumulate 
data on these processes. Me .;- POLIS data collection is 
completely unobtrusive. When teachers use the POLIS 
tools, their activity leaves records in the form of specific 
objects built for their classes, such as lessons. When 
students work within POLIS, their activity creates re- 



sponses archived within the course directory. Both 
instructors and students generate logs of use and of 
movements from page to page within a site. Additionally, 
new quiz and survey tools built within a related system 
(tentatively known as WILBUR) allow us to gather self- 
report data of any sort from students and instructors. 

The lessons constructed by POLIS instructors and the 
choices they reflect can be analyzed both quantitatively 
and qualitatively at any point in time. A search tool 
designed to allow instructors to locate relevant examples of 
each dialogue type also allows us to determine how 
frequently each lesson protocol is being chosen. These can 
be cross-classified with the type of subject being taught to 
build knowledge about the fit between protocols and 
subject matter. For example, the Virtual Peer is modelled 
explicitly after a protocol developed for the teaching of 
physics, but it is by no means limited to that field and has 
yet to be used in a science course at the University of 
Arizona. It has, on the other hand, been used extensively in 
social science courses, and it could be adapted to courses 
in any area. We can also examine the content of the 
protocols for internal evidence that instructors are attempt- 
ing to appropriate the structures for other than their 
intended purposes. We found early on that instructors in 
some fields were using the One-Minute Essay framework 
to post announcements and assignments, setting the 
number of prompts to “0” to suppress the production of a 
student response page. In effect, this means that instructors 
chose to make their web resources noninteractive. To 
discourage this way of using the protocols, we added 
separate tools for posting assignments and announcements, 
but we have not yet been successful in getting instructors 
in math and science fields to use online writing to support 
learning. 

Student responses and logs function similarly as 
sources of data. By examining and content analyzing 
student writings, we can recover surprising information on 
how students orient to the various features of the protocols. 
For example, we have discovered simply by examining 
automatically generated archives and logs that our original 
design for the Recitation protocol has the unintended effect 
of making the “model answer 7 ’ seem to be the point of the 
lesson. Stripped down to its structural minimum, a 
recitation has three required elements: a question, a 
candidate answer, and an evaluation. Our online imple- 
mentation of the recitation originally consisted of these 
three elements, with the answer taken as open-ended essay 
and the evaluation offered as a “model answer” returned 
when the students submitted their answers. In many 
situations, getting the “right” answer is of relatively little 
importance compared with having a chance to compare a 
candidate answer with an expert model. 

Unfortunately, we have found through examination of 
logs generated as byproducts of the students’ activity that 
they routinely skip to the model answer before giving their 
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own answer. Checking input to be sure the student has 
made an answer is useless, since students can simply enter 
a nonsense string. Adding a fourth step to the protocol in 
which students are asked for their reaction to the compari- 
son of their own answer to the model has also done nothing 
to fix this defect: students who have simply copied the 
model answer into the response space will assert proudly 
that their answers were “exactly right,” since they matched 
the model exactly. The lesson they are learning is “exactly 
wrong,” of course, and knowing this, we can tinker with 
the protocol to try to prevent it. 

Although surprising amounts of information can be 
recovered unobtrusively, from records generated automati- 
cally as byproducts of teaching and learning, we do also 
gather data systematically and overtly using online surveys 
and questionnaires. POLIS went into production in Fall 
1996. By Spring 1997 we had an end-of-semester question- 
naire for instructors to complete, reporting on their 
experience with POLIS. Unfortunately, the number of 
courses using the system was still rather small, and among 
these the response rate to the survey was low. Conse- 
quently, we learned relatively little from these question- 
naires compared to what we learned from the unobtrusive 
data. Much of what we learned from the questionnaire data 
could have been learned from direct examination of usage 
data: for example, instructors gravitated first to the 
information tools (webliography and bibliography), 
overlooking entirely the powerful interactive capabilities 
of POLIS. Not surprisingly, instructors found it easiest to 
adopt the “stage one” tools that closely resemble their prior 
teaching practices than the “stage two” tools that attempt 
to induce new practices (Hall, 1993). Survey data from 
Fall 1997 are not available at this time, but even without 
survey data a trend toward increasing use of the dialogue 
and discussion tools is evident. 

Assessment and database tools being introduced for 
Spring 1998 as part of the WILBUR initiative will greatly 
expand our ability to evaluate impact on student outcomes. 
These tools will enrich the authoring environment by 
adding the capability of building quizzes, tests, and 
surveys for courses and other uses, and by linking data 
drawn from responses to these instruments to POLIS 
archives and to other information systems. As with the 
current data sources in POLIS, these assessment tools will 
generate data as byproducts of teaching and learning. For 
example, a quiz used to assess a student’s comprehension 
of the material contained in an online lesson will serve 
secondarily to help us evaluate the effectiveness of the 
lesson design. 

Conclusion 

POLIS was designed to make teaching on the web as 
easy as possible and to work well within the ordinary 
preparation and planning practices of teachers. The 
authoring tools are available to instructors 24 hours a day 



from any point of access to the Internet, and they make 
instructors completely autonomous web publishers, 
independent of consultants, network administrators, 
programmers, or other support staff. Ease of use and 
ubiquitous availability to the instructor is crucial to the 
usefulness of the system as a platform for disseminating 
current knowledge about teaching and learning and as a 
device for gathering new data on what works. 

Instructors come to POLIS for the authoring tools; but 
in using the tools they learn about new teaching strategies 
and they contribute information automatically that helps 
refine our knowledge about teaching practice and learner 
outcomes. Students come to POLIS for the lessons and 
other resources; but in doing their work they too contribute 
data that we use for evaluation of the tools themselves. 

This interweaving of instructional support and instructional 
assessment allows us to build a self-correcting system 
whose development directions emerge constantly from 
examination of teaching practices and learning outcomes. 
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